feat(evaluate): grade a driver: docker row inside a container of its own image - #161

…s own image

Detached grading refused `driver: docker` outright, on the reasoning that
"grading cannot start a container (there is no agent to run in it)". The premise
was wrong. `DockerRunner` starts a container that runs `_run-task-internal`,
which builds an in-container Orchestrator — and a grade is that same Orchestrator
with `prior_result` set and no agent. The mechanism existed; it was never wired.

why it matters, measured
------------------------
`tasks/byod_smoke_test.yaml` asserts `test -f /opt/byod_marker`, a file baked
into its image. The IDENTICAL row:

    graded in a container ->  SUCCESS 1.000   (no stamp)
    graded on the host    ->  FAILURE 0.000   (graded_on_host: True)

The host is answering "is that marker on THIS machine", which nobody asked. That
is the general shape for any container task: its criteria address the image's
paths and toolchain. So the refusal was protecting against a real hazard by
making the feature unusable for the tasks that most need isolation.

the shape
---------
`_should_grade_in_container` -> `_grade_in_container` -> `DockerRunner(
prior_result=, grade_workspace=)`. The container gets TWO mounts, and their
separation is the whole design:

  * the grading pass's own fresh run_dir at CONTAINER_OUTPUT_DIR, whose task.json
    the host folds back into the row — preserving task.execute.json exactly as on
    the host path;
  * the executed workspace at CONTAINER_GRADE_WORKSPACE, read-WRITE and NOT a
    copy, adopted rather than written over. Read-write because criteria
    legitimately mutate what they grade; not a copy because that is what the host
    path already proved wrong (the template filter drops node_modules / dist /
    build / .venv, so a criterion reading those fails as a copying artifact).

The container half reuses the same `regrade_in_place` rather than restating it —
`evaluate`'s run-dir mode and `run --resume` drifted apart once already, and that
is why that function exists. It is driven by `context.json`'s `regrade` flag plus
a staged `prior.json`, both coerced on arrival: a truthy `"false"` would re-RUN
the agent over the workspace the operator asked only to grade.

A container-graded row carries NO `graded_on_host` stamp, so it is
indistinguishable from a `run` row. That parity is the point.
`--allow-host-grading` survives as the escape hatch (no docker here; criteria
known host-portable) and still stamps.

gated on the env var, never the driver
--------------------------------------
`IN_CONTAINER_ENV`, because the in-container entry point rewrites
`docker` -> `tempdir` before building its Orchestrator — a driver-based test
reads an already-changed value, and a grading container would dispatch a grading
container. That env var had four literal spellings; it now has one definition in
`models/container_paths.py`.

CE052 matched only the literal, so the migration made it read a constant-based
gate as NO gate — a rule instructing the author to paste the literal back,
arguing against the SSOT it exists to reinforce. It now accepts both spellings.
CE021 also caught the new `prior.json` parse; it degrades to a named message
rather than surfacing as "container exited without producing task.json".

verified
--------
End to end against real docker, both entry points:
  - `evaluate <run_dir>`: hello_world_docker 3/3 SUCCESS, byod 1/1 SUCCESS,
    no flag, no stamp, driver still recorded as docker, duration preserved to
    the digit, post_run run once in the grading phase.
  - `run --resume`: byod SUCCESS, exit 0, unstamped.
  - negative control: same byod row with --allow-host-grading -> FAILURE 0.000,
    stamped.

11 new tests (routing truth table incl. the recursion gate, dispatch payload,
both refusals, wire format, and controls asserting an ordinary run stages and
mounts neither).

ruff clean, pyright 0 errors, 455 lint rules pass,
5406 passed / 6 skipped at CI's -n 2, 92.54% coverage.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ed grading

Addresses the 8-axis review on #161: 6 blockers, 10 non-blocking findings and
7 nits. The two that change verdicts:

`recorded_task_file` was not threaded through the in-container regrade branch
(`_grade_recorded_run` / `regrade_in_place`), so every container-graded row
re-recorded `/work/task_dir/task.yaml` as its `source_file` — the exact defect
that parameter was added to fix, reintroduced one caller down, and enough to
make a later `evaluate <run_dir>` refuse or mount the wrong task dir.

The two known equivalence gaps are now STAMPED on the row, not only logged:
`graded_without_pre_run` and `graded_with_rebuilt_image`. `stamp_host_grading`'s
own docstring already gave the argument — a console warning does not travel with
task.json into run.json, the reports or the evalboard — and 3 of the 10 in-tree
docker tasks hit the pre_run case with no flags via `execute` -> `run --resume`.
The dockerfile_path warning the user guide promised did not exist; it does now.

Diagnosability and host safety on the grading container:
- logs are folded out of the scratch dir in a `finally`, not only on success.
  A failed container deleted `docker.log` while its own error said "See {path}".
  `grade.log` — a documented run-layout artifact — was never folded out at all,
  so a `driver: docker` row was the one shape a detached grade left without one.
- both log copies refuse a symlinked destination (`shutil.copy2` follows one;
  the sibling verdict write uses `write_text_atomic` for exactly that reason).
- the verdict write raises RegradeError, never a bare OSError: it sits outside
  the dispatch `try`, where `evaluate` let it escape into Typer after a
  SUCCESSFUL grade and `run --resume` reported a correct verdict as a failure.
- `grant_container_access` returns what it widened and `run()` restores it. The
  graded workspace is the caller's tree; an operator-supplied `--workspace` was
  left world-writable permanently.
- a container grade emits its own `CoderEval.Task.End`, mirroring batch.py. Every
  container runs `TELEMETRY_ENABLED=false` under "container silent, host emits
  once"; the grading path had inherited only the silent half.

Consent prompt:
- the dispatch renders as ONE command. Argv fragments were appended as separate
  entries, so one `docker build` was reported as "4 shell command(s)".
- it now names every host path exposed, not just `sandbox.docker.*`: the task
  directory copied from the recorded `source_file`'s parent, the auto-mounted
  plugins / template dirs / system_prompt_file, and the writable ~/.claude copy.
- `include_setup_phase` + `grade_in_place` collapse to one parameter. They were
  exact complements at every call site, with nothing rejecting the incoherent
  pairings — on a flag that gates a security disclosure.

Also: CE053 widened to the run-log filenames (docker.log was three unrelated
literals across two packages, and its consumer skips silently when absent);
`_quarantine_record` extracted from the two identical skew refusals; the stale
"instead of refusing" text corrected in both USER_GUIDE flag tables and the run
CLI help; the duplicated --allow-host-grading paragraph and mis-pointed "rely on
it" fixed; Rich markup escaped on the four handlers that render recorded strings;
the Sandbox/prior Optionals removed in favour of real narrowing; test doubles
bound to DockerRunner's real signature.

Tests: the in-container regrade branch is driven end to end (deleting either
`recorded_task` or `recorded_task_file` now fails, verified by mutation), plus
the failure-path log rescue, the symlink refusal, the OSError wrap, both stamps,
the one-command rendering, the host-path disclosure, telemetry parity and the
mode restore. 5447 passed, 6 skipped; ruff and pyright clean.

Not done, deliberately: pinning the resolved image by digest needs the RUN path
to record it first, so the row says the rebuild happened rather than the guide
claiming a control that does not exist.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…py + test_detached_grading_boundaries.py

CodeQL flagged 9 open alerts (all `py/import-and-import-from`, note severity)
introduced by the last commit's new test functions: each does a local
`import coder_eval.X as alias` for monkeypatching a module attribute, while
the same file also has a module-level `from coder_eval.X import (...)` for
the same fully-qualified module — confusing, per the rule's own rationale,
even though nothing here is a correctness bug.

Fixed by switching the local imports to the "from parent import submodule as
alias" form (`from coder_eval.isolation import docker_runner as dr`,
`from coder_eval.orchestration import regrade as rg`, `from coder_eval import
models`), which still binds the real module object — required for
`monkeypatch.setattr(dr, "DockerRunner", ...)` to affect the module's own
`from ... import DockerRunner` lookups at call time — without colliding with
the top-level `from coder_eval.X import (...)` imports of the same modules.

No behavior change: 109/109 tests in the two files still pass, ruff/pyright
clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>